A BizTalk Group consists of the set of databases
hosted in SQL Server. SQL Server provides high availability for BizTalk
Server 2006 applications through Windows cluster installations hosting
the BizTalk configuration and operation databases. For BizTalk
applications, the servers with BizTalk installed provide the runtime
environment with SQL Server as the persistent store. Therefore, BizTalk
Server 2006 backup, restore, and disaster recovery procedures are
heavily focused on procedures related to SQL Server.
A BizTalk Server
solution's availability and performance are highly dependent on
maintaining the SQL Server-based back end where the BizTalk Group is
located. Every message hat is received by a BizTalk host instance enters
the SQL Server Messagebox database, tracked to some degree, retrieved
from SQL Server by the host instance where the subscribing orchestration
is executed with the results put back into SQL Server for transmission
on a host instance send port. As you can see, overall BizTalk
performance is highly dependent on the underlying SQL Server
performance.
Changes to the
BizTalk Group SQL Server schema are not supported. Do not modify the SQL
Server schema for any tables or stored procedures that are part of the
BizTalk Group.
|
|
Luckily the BizTalk
product team provides several SQL Agent jobs to assist with keeping the
BizTalk Group running in top form. SQL Server Agent makes message bodies
available to HAT and WMI, and enables you to run jobs to clean up the
Messagebox databases.
The SQL Agent jobs are
created when the BizTalk Group is configured using Configuration.exe.
The user who runs Configuration.exe is also designated as the SQL Agent
job owner. The account designated as the job owner is the security
context that the SQL Agent job executes under. If any SQL Agent jobs are
failing, consider changing the job owner to a different account that
has the required privileges to folders or UNC shares, etc., so that the
job can complete successfully.
The SQL Agent jobs and a description of each are listed in Table 1.
Do not alter the schedule for any jobs except possibly for the Backup
BizTalk Server SQL Agent job. Details configuring the Backup BizTalk
Server SQL Agent job are covered in the subsection titled "Configuring
the Backup BizTalk Server SQL Agent Job."
Table 1 : BizTalk Group SQL Agent Jobs
SQL Agent Job | Remarks |
---|
Backup BizTalk Server (BizTalkMgmtDb) | Performs
backup operations on BizTalk databases that participate in Distributed
Transaction Coordinator (DTC) transactions. Not all BizTalk databases
are part of this job. Also, additional databases can be added to this
job. This job is disabled by default and must be configured in order to
run. Scheduled to perform full backup daily and log backup every 15
minutes by default. |
CleanupBTFExpiredEntries Job_<BizTalkMgmtDb> | New
in BizTalk Server 2006, this SQL Agent job cleans up expired BizTalk
Framework (BTF) entries in the BizTalk Management Database. Scheduled to
run every 12 hours and enabled by default. |
DTA Purge and Archive (BizTalkMsgBoxDb) | New
in BizTalk Server 2006, this SQL Agent job automatically archives data
in the BizTalk Tracking (BizTalkDTADb) database and purges obsolete
data. This job is disabled by default and must be configured in order to
run. Scheduled to run every minute by default. |
MessageBox_DeadProcesses_ Cleanup_<BizTalkMsgBoxDb> | Detects
when a running BizTalk host instance has stopped. This SQL Agent job
releases all in-progress work for that host instance so that it can be
picked up by another host instance. Scheduled to run every minute and
enabled by default. |
MessageBox_Message_Cleanup_<BizTalkMsgBoxDb> | Removes
messages that are no longer referenced by any subscribers in the
Messagebox database tables. This job is disabled by default, but it
should not be manually run. This job is started when needed by the MessageBox_Message_ManageRefCountLog_ <BiztalkMsgBoxDb> job. |
MessageBox_Message_ManageRefCountLog_ <BizTalkMsgBoxDb> | Manages
the reference count logs for messages. It determines when a message is
no longer referenced by a subscriber.Scheduled to run every minute and
enabled by default. |
MessageBox_Parts_Cleanup_<BizTalkMsgBoxDb> | Removes
message parts that are no longer referenced by any messages in the
Messagebox database tables. Scheduled to run every minute and enabled by
default. |
MessageBox_UpdateStats_<BizTalkMsgBoxDb> | New
in BizTalk Server 2006, this SQL Agent job manually updates database
statistics for the BizTalk Messagebox database. Scheduled to run every 5
minutes and enabled by default. |
Operations_OperateOn Instances_OnMaster_<BizTalkMsgBoxDb> | New
in BizTalk Server 2006, this SQL Agent job is required for
multipleMessagebox database deployments. It performs operational tasks
on Messagebox databases. Scheduled to run every minute and enabled by
default. |
PurgeSubscriptionsJob_<BizTalkMsgBoxDb> | Purges unused subscription predicates from the Messagebox database. Scheduled to run every minute and enabled by default. |
Rules_Database_Cleanup<BizTalkRuleEngineDb> | New
in BizTalk Server 2006, this SQL Agent job cleans up expired BizTalk
Framework (BTF) entries in the BizTalk Management Database. Scheduled to
run every 12 hours and enabled by default. |
CleanupBTFExpiredEntries Job_<BizTalkMgmtDb> | New
in BizTalk Server 2006, this SQL Agent job purges old audit data from
the Rule Engine database every 90 days. It also purges old history data
from the Rule Engine database every 3 days. Scheduled to run every hour
and enabled by default. |
TrackedMessages_Copy_<BizTalkMsgBoxDb> | Copies
message bodies of tracked messages from theMessagebox database to the
Tracking database. Scheduled to run every minute and enabled by default. |
Microsoft strongly
recommends monitoring the SQL Agent service and the individual jobs
using a monitoring tool such as MOM or similar enterprise monitoring
product. If any of these jobs starts failing, it is a strong indication
that there are performance issues with the application. It is also a
good idea to monitor how long it takes for the SQL Agent jobs to run,
perhaps on a weekly basis. If the jobs are taking longer and longer to
run over time, it is another indication that there may be performance
issues.
For completeness, the
BizTalk Server 2004 SQL Agent job TrackingSpool_Cleanup_
<BizTalkMsgBoxDb is no longer present in BizTalk Server 2006.
The next step is to configure the necessary SQL Agent jobs to perform backups and to maintain the BizTalk Group.
SQL Agent Job Configuration
The following jobs from Table 9-1 require configuration before they can be enabled and can run successfully:
NOTE
The SQL Agent job
MessageBox_Message_Cleanup_<BizTalkMsgBoxDb> is disabled by
default. It's not supported to enable this job or run it manually. This
SQL Agent job is managed by the
MessageBox_Message_ManageRefCountLog_>BiztalkMsgBoxDb> SQL Agent
job.
The following two subsections cover how to configure these SQL Agent jobs.
Configuring the Backup BizTalk Server SQL Agent Job
The Backup BizTalk
Server SQL Agent job is a critical job that must be configured in order
to be able to successfully back up the BizTalk Server 2006 databases
that participate in Distributed Transaction Coordinator transactions.
Databases that participate in DTC transactions such as with BizTalk must
be backed up and restored as a set to ensure consistency.
NOTE
Not all BizTalk
databases are backed up as part of the Backup BizTalk Server job.
Backing up these databases is covered in the next subsection.
The following databases are backed up as part of the Backup BizTalk Server SQL Agent job:
BizTalk Configuration (BizTalkMgmtDb)
BizTalk Messagebox (BizTalkMsgBoxDb)
BizTalk Tracking (BizTalkDTADb)
Rule Engine (BizTalkRuleEngineDb)
BAM Primary Import (BAMPrimaryImport)
Trading Partner Management (TPM)
HWS Administration (BizTalkHWSDb)
BizTalk EDI (BizTalkEDIdb)
These databases must
be backed up by the Backup BizTalk Server SQL Agent job and cannot be
backed up using the normal SQL Server backup procedures. The reason is
because BizTalk uses SQL Server log marks to keep the set of databases
consistent as part of DTC transactions. The Backup BizTalk Server job
creates a log mark and then backs up the database log for each database
that is part of the Backup BizTalk Server SQL Agent job. This log mark
is used when restoring the last log file for each database so that
transactional consistency is maintained. Here are the steps to configure
the Backup BizTalk Server SQL Agent job:
In SQL Server 2000 Enterprise Manager or SQL Server 2005 Management Studio, navigate to the SQL Agent jobs list.
Right-click Backup BizTalk Server (BizTalkMqmtDb) and select Properties.
In the Job Properties dialog under Select a page, click Steps to view the job steps.
In the Job step list, click BackupFull, and then click Edit.
On
the General page, in the Command box, replace '<destination
path>' with the full path (the path must include the single quotes)
to the computer and folder where you want to back up the BizTalk Server
databases. Also add a new parameter by typing a comma and then a number
one (",1") at the end of the parameter list for the stored procedure sp_BackupAllFull. Adding this parameter enables an automatic full backup after a backup failure. Click OK when finished.
NOTE
The default
frequency for the BackupFull job is d for daily. Other values are hourly
(h/H), weekly (w/W), monthly (m/M), or yearly (y/Y). The first time the
job is run during a new period, a full backup is performed. Also, the
default name is BTS, which will be part of the backup file name. Change
this to reflect a better name for the application such as OrdSys for an
application named Order System.
In the Job step list, click MarkAndBackupLog, and then click Edit.
On
the General page, in the Command box, replace '<destination
path>' with the full path (including single quotes) to the computer
and folder where you want to store the BizTalk Server database logs and
then click OK. The <destination path> may be local or a UNC path
to another server.
NOTE
We recommend a UNC
share to store the backup files on a different file system than where
the databases reside for production environments. For a dev or test
environment, if you are not concerned with maintaining offsite backup
sets or multiple backup sets, you can consider using a local path
instead of a UNC path.
Also, for the job step MarkAndBackupLog, Log Mark Name is part of the naming convention for backup files:
<Server Name>_<Database Name>_Log_<Log Mark Name >_<Timestamp>
Replace "BTS' with a more appropriate name for the solution.
In the Job step list, click Clear Backup History, and then click Edit.
On
the General page, in the Command box, change DaysToKeep=<number>
to the umber of days (default is 14) you want to keep the backup
history, and then click OK twice to close the Job Properties dialog box.
NOTE
The DaysToKeep setting
is not related to how many sets of backup files are maintained. Backup
file sets must be handled manually by copying to another system for
long-term archival.
Change
the backup schedule for MarkAndBackupLogSched if desired and then
right-click the Backup BizTalk Server SQL Agent job and select Enable.
The default schedule is to perform a log backup every 15 minutes.
Once the Backup
BizTalk Server job is configured and enabled, right-click and select
Start Job to test. Click F5 to refresh the status on the Jobs node. If
the result is not successful, check the following:
Verify that the destination folder exists and is reachable if a UNC share.
Check that the job owner has permissions on the destination folder.
Ensure that linked servers are configured properly if BizTalk databases are present in multiple SQL Server database instances.
For SQL Server
2005, there are additional security settings for linked servers. When
configuring linked servers in SQL Server 2005 as part of the Backup
BizTalk Server SQL Agent job, click the Security tab and select the "Be
made using the login's current security context" option. Next click
Server Options, set RPC Out to True, and then click OK.
|
|
Be aware that the file
name includes the date/time from when the backup file was created. This
date/time is GMT time, not local time. If you look at the Date Modified
field in Windows Explorer, you will see the local time.
Also, the Backup
BizTalk Server SQL Agent job does not manage disk space, meaning it will
continue to copy files into the same directory until the drive runs out
of space. This allows the administrator to decide how many backup sets
to keep on disk as well as how many to archive to an offsite location,
deleting the files from disk after archival.
The next subsection covers how to configure the DTA Purge and Archive SQL Agent job.
Configuring the DTA Purge and Archive SQL Agent Job
With companies having
to comply with IRS, general accounting, and legislative requirements for
business reporting, BizTalk Server provides extensive tracking
capabilities to help with complying with these mandates. This data must
be kept for various periods of time to meet reporting requirements.
BizTalk Server 2006 adds the DTA Purge and Archive job to help automate
the backup of tracking data including the ability to perform on-the-fly
validation of tracking data backups using another instance of SQL Server
to ensure that a complete record of activity is maintained and
available.
In addition to providing
data archival, the DTA Purge and Archive SQL Agent job performs data
pruning to help keep the system running smoothly. As with any database
system, unchecked growth in table size will eventually push the limits
of the hardware. In general, there are two solutions to this problem,
buy more disks or a faster disk or have a purge and archival policy to
"prune" the databases where it makes sense. While all database-based
systems benefit from more and faster disks, BizTalk has a process to
keep the BizTalk Tracking and Messagebox databases performing optimally
by automating purging and archival tasksthrough the DTA Purge and
Archive SQL Agent job.
The DTA Purge and Archive
job purges various tracking information such as service instance and
message information, orchestration event information, and rule engine
tracking data. The purge process is based on the age of the tracking
data, which is maintained by having a time stamp added when tracking
information is inserted into the database. The DTA Purge and Archive job
has a soft purge and hard purge
process. The soft purge processescompleted instances, while the hard
purge processes incomplete instances. Note that both soft purge and hard
purge process just the tracking data, not the actual running instances,
so they have no effect on actual data processing data. The purge
process helps to optimize tracking processes and HAT operations when
looking at historical data. Here are the steps to configure the DTA
Purge and Archive SQL Agent job:
Depending on whether you are on SQL Server 2000 or SQL Server 2005, navigate to the Management node and view the SQL Agent jobs.
In the details pane, right-click DTA Purge and Archive (BizTalkDTADb), and then click Properties.
In the Job Properties dialog box, click the Steps tab, click Archive and Purge, and then lick Edit.
On the General tab, in the Command box, edit the following parameters as appropriate, and then click OK.
NOTE
For the soft
purge, the sum of LiveHours and LiveDays is the live window of data that
will be maintained for the BizTalk Tracking database. All tracking data
associated with completed instances older than the live window will be
deleted and archived.
@nLiveHours tinyint: Default is 0 hours.
@nLiveDays tinyint: Default is 1 day.
@nHardDeleteDays tinyint: Default is 30 days.
@nvcFolder nvarchar(1024): Specify the folder or UNC share to put the tracking data backup files.
@nvcValidatingServer: SQL Server instance where validation is performed. Default is null.
@fForceBackup int: Default is 0. This is not currently implemented.
Here is an example command that specifies that soft purge occurs every 12 hours and hard purge occurs every 7 days:
exec dtasp_BackupAndPurgeTrackingDatabase 12, 0, 7, '\\BizTalkBackupServer\data',
null, 0
In the preceding example,
we left the validation server value as null; however, we recommend that
you set up a validation server for the tracking data to ensure that the
backup files of the tracking data for reporting and compliance purposes
are valid. Also, the data can be ueried on the validation server,
offloading potentially long-running queries from the production BizTalk
databases. To configure a validation server for the DTA Purge and
Archive SQL Agent job, you must have a separate instance of SQL Server
available. Having a validation server requires that the @nvcFolder
variable in the DTA Purge and Archive job points to a UNCshare
reachable by the validation server. The SQL Server instance where the
BizTalk databases are configured cannot also act as the validating
server. On the server designated as the validation server, perform these
steps:
In
either ISQL on SQL Server 2000 or in SQL Management Studio on SQL
Server 2005, open a file to execute a SQL file. Connect to the SQL
instance that is the validation server.
Select File =>
Open and then browse to this SQL script on the server/drive where
BizTalk Server 2006 is installed: \Program Files\Microsoft BizTalk
Server 2006\ Schema\BTS_Tracking_ValidateArchive.sql.
Execute the query to create a SQL Agent job called ValidateArchive on the validating server.
Open
either Enterprise Manager for SQL Server 2000 or SQL Management Studio
for SQL Server 2005 to set up the required linked servers. Linked
servers must be created between the following:
Each
of the BizTalk Messagebox (BizTalkMsgBoxDB) SQL Server instances and
the BizTalk Tracking (BizTalkDTADb) SQL Server instances. The SQL
instance hosting the DTA database requires a linked server to each SQL
instance hosting a BizTalk Messagebox and vice versa.
The
BizTalk Tracking (BizTalkDTADb) SQL Server instance and the validating
server SQL Server instance. Create a linked server on each SQL instance
to the other SQL instance so that the SQL instance hosting the DTA
database has a linked server to the validating server and vice versa.
Next, we turn our attention to monitoring best practices for the SQL Agent jobs.
Monitoring the BizTalk Group SQL Agent Jobs
Because the SQL
Agent jobs are critical to maintaining BizTalk performance, the jobs
must be monitored so that operations personnel can be alerted if a job
fails. The Microsoft SQL Server Management Pack contains MOM rules for
monitoring SQL databases, SQL Server Agent jobs, etc., for comprehensive
monitoring of SQL Server items. The BizTalk Server 2006 Management Pack
for Microsoft Operations Manager 2005 includes two rules, disabled by
default, for monitoring the health of two of the most important BizTalk
SQL Server Agent jobs. The rule namesas defined in the Management Pack
are
To monitor all
BizTalk Server SQL Server Agent jobs from within the BizTalk Server 2006
Management Pack, enable these rules and create additional rules for
other jobs that you want to monitor. To enable these rules, perform the
following steps in the MOM Administrator Console:
Create a copy of the two rules just listed in the BizTalk Server Core Rule group and rename each rule appropriately.
In
the criteria section for the MOM rule, change the wildcard comparison
for Parameter 1 because the job names are specific to the BizTalk Group
configuration.
You need to add the SQL
Server computers into the BizTalk Server 2006 Computer Group in MOM.
This is because the MOM rule needs to be evaluated on the SQL Server
computer, and the SQL Server computer will not be recognized as a
BizTalk Server computer unless BizTalk and SQL Server happen to be
installed on the same machine.